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ABSTRACT 

In response to a state-wide request in 1986-87 from 
the governor of Missouri, public institutions of higher education 
undertook assessment projects related to their students' basic 
skills. English faculty at Missouri Western State College conducted a 
composition assessment project in which they used timed hollstlcally 
scored writing as a means of placing students. Results were 
overwhelmingly negative. After three years of using the Writing 
Placement Exam (WPE) , however, valuable modifications have been made, 
including streamlining the administration of the WPE. It has been 
argued that the greater the consequences of error in assessment, the 
greater the responsibility of assessors to be humane, fair, and 
effective. With this in mind, great caution must be exercised when 
viewing hollstlcally scored timed writing as a curriculum evaluation 
tool. Timed hollstlcally scored writing shows the greatest promise as 
a placement instrument. If used wisely, it could produce benefits for 
faculty, students and education that outweigh the negative aspects. 
(PRA) 



a******************************************************************** 



* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 



CO 
CO 

CO 

Q 



Jane Frick and Karen Fulton 

Department of English, Foreign Languages and 

Journalism 
Missouri Western State College 
St. Joseph, MO 64507 

Paper delivered at the 1991 4 C's Meeting: March21, 1991 

Promises Kept and Broken: Holistically Scored Impromptu 

Writing Exams 

During the 1986-87 academic year Missouri's Governor Ashcroft 
requested that public institutions of higher education undertake 
assessment projects related to their students' basic skills. Partially as a 
response to this state-wide request, but also because of a need to 
evaluate the effectiveness of the placement procedures and 
instructional content of its composition courses, English faculty at 
Missouri Western State College conducted a composition assessment 
project, 1987-89. 

The composition faculty had a high degree of administrative 
support and, in general, were willing to explore the use of a timed 
holistically scored writing as a means of placing students. There was a 
high level of dissatisfaction with the present placement method- -the 
use of the ACT English score as a placer. Therefore, we anUcipated a 
fruitful, positive experience. Instead we received impressively negative 
results. 
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Promises Broken: Using impromptu writings to evaluate curriculum: 

Our plan was simple. As one means of evaluating the 
effecUveness of our curriculum, we decided to compare students' 
English skills before beginning a composition course to their skills 
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following completion of the college's composition sequence. Our 
college administration requested we pre- and post-test with a 
nationally-normed English exam. We opted for the English porUon of 
the ACT since pre- test scores would be available without additional test 
administration because the ACT is required for admissions. But 
because the EACT is an objective test requiring no writing on the test 
taker's part and because we believed "writing assessment must entail 
the use of direct measures of actual student writing" [cWa Position 
Paper], we decided to develop our own instrument— a 45-minute 
impromptu descriptive essay developed locally and scored holistically 
by English staff members who followed guidelines advocated by Edward 
M. White in Teaching and Assessing Writing. 

This Writing Placement Exam (WPE) would be given as a pre- and 
post-test to measure improvements in writing skills. Written by 
students before they register, we could also use the WPE score to place 
students in their beginning composition courses. At the end of our 
composition sequence, (for three semesters-spring, 1988, fall 1988, 
and spring 1989) we administered the EACT and the WPE in a single 
two-hour session during finals week to students completing their last 
required composition course. We scored both versions of each 
student's Writing Placement Exam during finals week in order to 
insure reliability. 

Even before the first 3f the end-of-testing semester testing 
sessions, the perils of using our holistically-scored impromptu essay to 
evaluate student improvements in English (and therefore composition 
curriculum effectiveness) became apparent. Some composition faculty 
had not read memos and /or listened at departmental meetings when 
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the assessment project with the special testing sessions were 
discussed. When their students learned late in the term that they had 
to take the tests during an extra session in exam week, students 
complained (without success) to administrative officials. Some 
instructors cancelled their classes' traditional final exam; others 
required that students return and sit for it. Some instructors told 
students that their score results on their second writing of the WPE 
would be factored into their final course grades; others told their 
students that the test was a waste of time but that they had to show up 
and write the exam anyway. In other words, some students were 
highly motivated to perform well on the WPE; other students lacked 
motivation and resented having to take the time during finals week for 
post-tests in the assessment project. 

We made a critical error in our zeal to report WPE score results 
to departmental faculty following the first testing session. We set up 
our data base listing students by composition sections so that as soon as 
we keyboarded pre- and post-test WPE scores at the testing site, the 
software not only churned out the change in individual scores and 
computed score ranges, standard deviations, and mean scores on pre- 
and post-W?Es for all students tested but also provided the ranges, 
standard deviations, and mean scores for each composition section. 
Not surprisingly, when the papers were scored, students from sections 
where teachers reported that the exam results "counted," tended to 
have higher scores on their second writing of the WPE than students 
from sections where the exam did not affect their course grade. 
Faculty compared their students' results with the results of other 
faculty's students; morale plummeted. At least one faculty member 
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went to the dean for reassurance that his students' overall negative 
performance on their second writing of the WPE would not be 
reflected in his annual evaluation. We learned never to report student 
data in any manner in which faculty comparisons could be made. 

Our WPE post-test results each semester yielded the same 
results. Students who scored low (below six out of a possible twelve 
points) on their first writing of the essay and who began their 
composition sequence in our development writing course tended to 
show significant increases (a mean increase of almost three points) in 
their scores on their post-WPEs. Students who scored above six on 
their first writing of the essay and who began the composition 
sequence in the first regular course showed a sustained rate of test 
scores on their post-test WPE. We believe two factors contributed to 
this lack of improvement. (1) Student scores regress to the mean; 
students who were at the high end (above nine) on the twelve-point 
scale for their pre-WPE were likely to score at or below their original 
score on the post-WPE. (2) The WPE, which asks students to write a 
persuasive essay, assesses students' ability to perform successfully in 
our first regular (nondevelopmental) composition course. Once that 
level is achieved, little growth should be expected. 

We concluded that the holistic scoring of our Writing Placement 
Examination as a post-test was not a valid means to measure students' 
improvement in English skills. While we evaluated students' actual 
writing rather than their responses to multiple-choice questions about 
writing, the second WPE only measured students' ability to write an 
impromptu 45-minute essay and did not test the kinds of reflective, 
analytical, and research skills we taught in our two regular composition 
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courses. Furthermore by using holistic scoring, we reduced student 
writing to a single, quantifiable number which allows for statistical 
manipulation but establishes opportunities for gross generalizations and 
inaccurate portrayals of student abilities and teacher performance. 

Our two-year writing assessment project has left us with a 
healthy skepticism about measuring students' improvement in English 
following completion of their composition sequence. Two sets of pre- 
and post-test scores on the WPE and EACT tell us little about our 
students or our composition courses. 

Promises Kept: Holistically-Scored Placement Exams 

Is the timed, holistically-scored essay, unproductive as a means 
of assessing writing? The Missouri Western experience would 
indicate the answer is "not entirely." Two distinct gains came from 
the experience. Recognizing that a single, timed-writing sample does 
not "measure" the writing skills of students leaving our composition 
classes, we now require our students to keep a portfolio. In these 
themes, letters, reports, poems, autobiographies, and term papers are 
the real evidence of our students' improvements and the reflection of 
our composition curriculum. Secondly, the timed holistically scored 
writing sample- modified to be sure-has found its greatest success as a 
measure of placement of incoming students. 

The use of the WPE provides us with a useful and, we believe, 
much more humane, fair, and effective means of placing students in 
our composition sequence than the use of a single EACT score. Prior 
to 1987, students at Missouri Western were placed solely on the basis 
of their EACT-a score of 15 and below placed a student in 
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developmental composition; a score of 16 and above placed a student 
in the first "regular" course. Faculty in the department complained 
that the courses sorted this way were extremely uneven in terms of 
writing ability. In addition, over 20% of our students arrived on our 
campus with no EACT available, necessitating their waiting at least a 
semester to begin their work in composition or beginning 
automatically in the developmental course. Given our population base 
with its large non-traditional student component (and concurrent high 
anxiety), the EACT scores were also seen as non-reflective of what a 
student would be able to do within the classroom. 

There were some negative "start-up" costs. Our initial 
administration cost approximately $1000 for supplies. Because the 
original WPE was part of a grant in assessment from the college, the 
personnel costs initially were covered by the grant. The grant is now 
concluded, and the Department of English, Foreign Languages, and 
Journalism has had to absorb the costs of the WPE into its budget. 
Financial support has come to us from the Division of Liberal Arts and 
Sciences, and gradually, the perception is growing that the WPE is a 
service to the school, not just to the department. Our President and 
Academic Vice President have supported the use of the WPE by scoring 
for us; we hope that their continued involvement will result in 
continued funding of supplies and sUpends for readers. 

There were some negative "emotional" costs as well. Change can 
be threatening to faculty— college wide. While most faculty at the 
college applauded our efforts, there were some who wished for a more 
"objective," less "subjective" measurement of incoming students' 
wriUng abilities. Because we tested "high" EACT students into our 
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developmental course (as well as "low" EACT students into our regular 
course), we were perceived as placing everybody in the lower English 
course. When the figures continually showed that the same number 
would have been placed in the two courses regardless of the method 
used, some of these objections died away. When such objections are 
raised by faculty, our method has been to invite the speaker to become 
a scorer at the first opportunity. 

Our three years of using the holistically scored WPE have 
produced valuable modifications. Two years ago, we began to give a 
two-point bonus to students who scored at the 80th percentile or 
higher on the EACT to insure that a high EACT student who tested 
into the developmental course would have written a paper at least in 
the lower quarter of the sample. We have streamlined our 
administration of the WPE. On the day a new student arrives, we now 
screen her for EACT, test her in a standardized situation, score her 
exam, and distribute the results to her before she goes on to select her 
classes for the semester. During the five summer Student Orientation 
Programs in 1990 we tested 738 students (approximately 150 per 
program). 

This year we produced a brochure on the WPE which described 
testing and placement procedures and which announced the topic of 
the timed essay. These are mailed to students when they are invited 
on campus for orientation sessions. We believe providing students the 
WPE brochure helps them to reduce writing anxiety and allows those 
students who prefer to prewrite and think about the topic to do so. In 
addition, student who feel they have been misplaced as a result of their 
WPE results may request a conference with the Coordinator of the 
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Writing Placement Examination. The goal in each of these conferences 
is to reassure the student and to help the student understand why the 
placement was made. Each individual case, therefore, is dealt with 
individually. Sometimes the student will retest; sometimes the student 
will be told to request a new placement after a week of classes; 
sometimes a student will be requested to produce a portfolio of her 
work. We feel that the WPE is an indication that we as a department 
and as a school care about the students, not that we are out to punish 
them. 

The numbers speak very positively. Had we placed by EACT 
alone in the fall of 1990, we would have placed four more students in 
the lower English classes than we did by using our holistically scored 
WPE. However, we moved nearly 300 entering freshmen. The 
numbers of students in each course remained substantially the same, 
but the people were not the same. Our faculty tell us that the classes 
are easier to teach. We have heard anecdotally from teachers in both 
beginning writing courses that the placement achieved through the 
holistically- scored timed writing gives them classes comprised of 
students able to write on a corresponding level. 

Two other positive benefits must be mentioned. The scoring of 
the WPE has created an opportunity for faculty from across the 
curriculum and administrators to come together to discuss writing and 
to evaluate the writing of our students. Again and again, we have found 
that English teachers are less critical of student writing than those 
from outside the discipline. As non-English faculty and administrators 
see the general level of writing as it enters our college, a positive 
atmosphere is created. Secondly, by choosing to evaluate a student's 
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writing rather than their ability to recognize error as evidenced by the 
EACT, we are strongly reinforcing writing as a higher order cognitive 
skill. 

Our goal as assessors and teachers of writing must be to use 
writing tasks and evaluations which are governed by the definition 
assidere-to sit beside. Dr. Richard Larson, at the 1989 Missouri 
Colloquium on Writing Assessment argued persuasively that the greater 
the consequences of error in assessment, the greater is our 
responsibility as assessors to be humane, fair and effective. Or as 
Stephen Jay Gould says "Few tragedies can be more extensive than the 
stunting of life, few injustices deeper than the denial of an opportunity 
to strive or even to hope, by a limit imposed from without, but falsely 
identified as lying within" (29). With this in mind, we must view a 
holistically- scored timed writing as a barrier or as an evaluation tool of 
curriculum with great caution. The greatest promise of the timed 
holistically scored writing is as an instrument for placement, which, if 
used wisely, can produce benefits for faculty, for students, and for 
education which outweigh the negatives. 
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